Unsupervised alignment for text to speech synthesis using neural networks

ABSTRACT

Generation of synthetic speech from an input text sequence may be difficult when durations of individual phonemes forming the input text sequence are unknown. A predominantly parallel process may model speech rhythm as a separate generative distribution such that phoneme duration may be sampled at inference. Additional information such as pitch or energy may also be sampled to provide improved diversity for synthetic speech generation.

BACKGROUND

Speech synthesis is generally modeled in an autoregressive manner, wherea statistical model is used to generate output speech based on an inputtext sequence. These models predict different phoneme lengths for theinput text sequences, but a single poorly predicted audio frame may leadto additional errors throughout the entire sequence of synthesizedspeech. Autoregressive models also scale poorly, especially as sequencelengths increase. Moreover, attempts to integrate autoregressive modelsinto parallel architectures have developed their own problems, such astrouble with audio-text alignment. Furthermore, autoregressive modelsmay lack diversity in synthetic speech outcome, where an input textsequence leads to a similar output each time a model is executed, whichmay be undesirable in many applications.

BRIEF DESCRIPTION OF THE DRAWINGS

Various embodiments in accordance with the present disclosure will bedescribed with reference to the drawings, in which:

FIG. 1 illustrates an example of a pipeline for speech synthesis,according to at least one embodiment;

FIG. 2A illustrates an example of a training pipeline for speechsynthesis, according to at least one embodiment;

FIGS. 2B and 2C illustrate an example of an inference pipeline forspeech synthesis, according to at least one embodiment;

FIG. 3 illustrates examples of alignment attention matrices, accordingto at least one embodiment;

FIG. 4A illustrates an example phoneme distribution, according to atleast one embodiment;

FIG. 4B illustrates an example pitch distribution, according to at leastone embodiment;

FIG. 5 illustrates an example environment for speech synthesis,according to at least one embodiment;

FIG. 6A illustrates an example flow chart of a process for generatingsynthetic speech, according to at least one embodiment;

FIG. 6B illustrates an example flow chart of a process for generatingsynthetic speech, according to at least one embodiment;

FIG. 6C illustrates an example flow chart of a process for training atext to speech system, according to at least one embodiment;

FIG. 7 illustrates an example data center system, according to at leastone embodiment;

FIG. 8 illustrates a computer system, according to at least oneembodiment;

FIG. 9 illustrates a computer system, according to at least oneembodiment;

FIG. 10 illustrates at least portions of a graphics processor, accordingto one or more embodiments; and

FIG. 11 illustrates at least portions of a graphics processor, accordingto one or more embodiments.

DETAILED DESCRIPTION

Approaches in accordance with various embodiments provide systems andmethods for end-to-end text-to-speech (TTS) models. In at least oneembodiment, TTS models are parallel or at least partially parallel.Various embodiments may include models that further model speech rhythmas a sampleable distribution to facilitate variable token durationduring inference. In at least one embodiment, systems and methodsprovide online extraction of speech-text alignment.

Various embodiments relate to systems and methods for TTS generationusing a predominantly parallel end-to-end model. Embodiments includeunsupervised alignment using a probabilistic determination in order toalign individual phonemes, that may have variable lengths, withcorresponding text. A generative model is used to sample over adistribution of plausible phoneme durations, where a boundary may be setto eliminate consideration of durations that are unlikely orunrealistic. Additionally, another generative model may be developed forpitch and energy of different phonemes, which may be extracted duringthe training phase. Diversity may be achieved by sampling diverselocations from the same probabilistic distribution. Systems and methodsmay also be directed toward improvements in training using distributionaware data augmentation where synthesized data may be tagged/categorizedduring training and then may be ignored or otherwise not used atinference time.

Various embodiments of the present disclosure are directed towardovercoming problems associated with traditional alignment tools withTTS, where these tools attempt to extract the alignment prior toinferences or rely on attention mechanisms. Using a forced aligner mayhave limited capabilities, for example, where an aligner is notavailable for a particular language or alphabet. Additionally, attentionmechanisms for aligners have proven unstable and may not converge tomeaningful solutions. Systems and methods of the present disclosure mayaddress these, and other problems, by extending attention-basedmechanisms to add a prior distribution, which in one or more embodimentsmay be cigar-shaped. Furthermore, systems and methods may deploy varioustechniques related to Hidden Markov Methods (HM), such as forward-subalgorithms and viterbi, in order to identify most likely strings of textfor given signals.

Embodiments may also address problems associated with insufficient orsmall data sets used for training. With respect to TTS synthesistraining pipelines, augmented data sample are rarely incorporatedbecause these samples may be considered as outside of a desireddistribution for modeling because, in many instances, it is undesirableto synthesize text to sound similar to the augmented data samples. In atleast one embodiment, systems and methods may incorporate dataaugmentation to improve generalization of models without having effectsthat bleed into the inference results. For example, augmented data maybe labeled or otherwise identified within one or more generateddistributions, where sampling during inference avoids those areas.

Diversity with synthesized speech is also addressed using systems andmethods of the present disclosure. In at least one embodiment, the sametext may be synthesized multiple times with each outcome being plausibleand qualitatively different. A generative model (normalizing flow) maybe utilized to sample plausible phoneme durations at inference time,which may provide improved diversity because one significant variationwith respect to speech is phoneme duration. Systems and methods may alsobe directed toward pitch and energy modeling, thereby providing greateroptions for diversity with synthesized speech. In at least oneembodiment, pitch and energy may be modeled and a probabilisticcomponent may sample from these models when generating output speech.

Various embodiments may also improve speaker conditioning variables. Forexample, traditional multi-speaker TTS models may assign a dimensionalvector for a particular speaker and then apply that vector whengenerating an output. However, using a single vector for speakers maylimit a model's ability to generate similar or different soundingoutputs without modeling individual variances for speakers. Embodimentsmay generate a vector as a function of an input and model across aspeaker's distribution at inference time.

A text-to-speech (TTS) pipeline 100, which may also be referred to asspeech synthesis, is illustrated in FIG. 1 that includes an input 102that may correspond to a textual input. It should be appreciated thatthe input may be an initial text input, such as an input provided by auser, a converted text input, such as an utterance that has beenevaluated and then converted to text, a sequence of text extracted froman input image or video, or the like. In at least one embodiment, theinput 102 may be responsive to a question or comment provided by a user,such as a conversational artificial intelligence (AI) system thatprovides answers responsive to user queries, among other applications.The illustrated input 102 may be formatted for inclusion within aprocessing framework 104, that may include one or more trained machinelearning systems to evaluate the input 102 for one or more features,which may enable conversion of the input 102 into an audio output thatemulates human speech.

In this example, the processing framework 104 includes a naturallanguage understanding (NLU) system 106, a prosody model 108, and a TTSmodule 110. As will be appreciated, the NLU system 106 may be utilizedwith one or more conversational AI systems to enable humans to interactnaturally with devices. The NLU system 106 may be utilized to interpretcontext and intent of the input 102 to generate a response. For example,the input 102 may be preprocessed, which may include tokenization,lemmatization, stemming, and other processes. Additionally, the NLUsystem 106 may include one or more deep learning models, such as a BERTmodel, to enable features such as entity recognition, intentrecognition, sentiment analysis, and others. Furthermore, the NLU system106 may enable conversion of linguistic units of the input 102 intophonemes, which may then be assembled together using the prosody model108.

In at least one embodiment, TTS model 110 may take a text responsegenerated by the NLU system 106 and change it to natural-soundingspeech. It should be appreciated that, in various embodiments, theprosody model 108 may be part of the TTS model 110. The output from theNLU system 106 may undergo various processes associated with the TTSmodel 110, such as linguistic analysis, synthesis, and the like.Additionally, parts of speech may be tagged. In various embodiments,output may be further analyzed for refining pronunciations, calculatingthe duration of words, deciphering the prosodic structure of utterance,and understanding grammatical information. Additionally, text may beconverted to mel-spectograms for output to the vocoder 112 to generatenatural sounding speech. As noted above, it should be appreciated that,in various embodiments, the vocoder 112 may be incorporated into the TTSmodel 110. Accordingly, an audio output 114 is generated that soundslike human speech.

Speech synthesis may be modeled sequentially in a fully autoregressivemanner, where training and interference speeds are unable to orinefficiently scale as sequence length increases. Additionally, errorswithin one or more audio frames may be propagated to other parts of thesynthesized speech. Parallelizing speech synthesis is desirable, but maysuffer from problems associated with phoneme length. For example,without synthesizing the phonemes, it is difficult to know theirindividual lengths. Additionally, methods that may first determinephonemes in input text, and then sample from a mel-spectogram, miss outon full end-to-end parallelization of systems and methods of the presentdisclosure. Systems and methods of the present disclosure may bedirected toward an end-to-end system that includes online alignmentalong with implementation of a generative model for inferring diverseresults.

Existing alignment techniques are often insufficient or unusable inparallel architectures. For example, existing techniques may distillattention from an autoregressive model into a parallel architecture thatincorporates a two-stage process, which may be costly to train.Moreover, these techniques are often limited, with independent trainingbeing used for individual languages or alphabets used with the system.Furthermore, these techniques often display a loss in inferencediversity, such as variability with speech rhythm. Systems and methodsmay overcome these drawback by incorporating one or more generativemodels, which may be separate models, for token durations.

A training pipeline 200, shown in FIG. 2A, may be utilized to constructa generative model for sampling mel-spectrograms given textual input andspeaker information. In this example, a mel-spectrogram 202 is providedwith a speaker vector 204 (0, which may encode speaker-specificcharacteristics, for augmentation 206. For example, the mel-spectrogrammay be for an audio clip of human speech, which may be represented as amel-spectrogram tensor X∈

^(C) ^(mel) ^(×T), where T is the number of mel-frames over the temporalaxis, and C_(mel) is the number of bands of dimensions per frame. Thespeaker vector 204 may include information for corresponding changeswith respect to a data distribution. In this example, augmentation 206may be applied with a certain probability, which may change themel-spectrogram 202 with corresponding changes to the speaker vector204, and then an output is directed toward a training pipeline 208. Itshould be appreciated that the mel-spectrogram 202 may also be directedtoward the training pipeline 208 without augmentation along with inputtext 210, which may be represented as a tensor of an embedded textsequence, as shown by ϕ∈

^(C) ^(txt) ^(×N), where N is a length. The resultant output is amaximum likelihood estimation (MLE) 212 over a latent space. This outputmay correspond to latent random variables for mel and durations (speechrate) that have been optimized.

An inference pipeline 250, shown in FIG. 2B, may sample from the MLE 212in order to generate an input sample 252. Random sampling of the MLE 212may enable diverse inference results. In at least one embodiment, sample252 is processed by an inference pipeline 254 to output a secondmel-spectrogram 256, which may then be processed to produce an outputaudio clip.

Various embodiments of the present disclosure enable sampling for bothmel-spectrogram frames and their durations at inference time whilemaintaining a parallel architecture for modeling. In operation, atemporal alignment is developed between the audio clip (e.g.,mel-spectrogram 202) and the text (e.g., input text 210). The alignmentmay be represented as A∈

^(N×T). Accordingly, a conditional distribution may be represented byEquation (1).P(X,A,F ₀ ,E|Φ,ξ)=P _(mel)(X|Φ,ξ,A,F ₀ ,E)P _(dur)(A|Φ,ξ)P _(pitch)(F₀|Φ,ξ)P _(energy)(E|Φξ)   (1)

As noted, in the conditional representation, X represents amel-spectrogram tensor, A represents the alignment, F₀ represents pitch,and E represents energy. Accordingly, as will be described below,various embodiments may enable sampling over a variety of differentdistributions, which may include duration, pitch, energy, and otherproperties of speech not captured by duration, pitch, and energy, suchas, but not limited to, intonation, stress, tempo, rhythm, and the like

It should be appreciated that, in various embodiments, one or moreportions of instructions (e.g., software instructions) for executing atleast a portion of Equation (1) may be parallel. By way of example,P_(mel) may be parallel. However, one or more other parts, or componentsof parts, of Equation (1) may include one or more autoregressivecomponents, such as duration, pitch, energy, and text. Accordingly,different components of various embodiments may balance costly trainingsteps.

In operation, normalizing flows are applied to mel-coding in TTS.Distributions may be modeled such that each time step in a distributioncan be sampled from a simple distribution. In various embodiments,independent and identically distributed random variables are assumed.Accordingly, the MLE 212 with respect to data samples x may berepresented by Equation (2)log p _(x)(x)=log p _(z)(g ⁻¹(x))+log|det J(g ⁻¹(x))|,  (2)

where p_(x)(x) represents the unknown likelihood function for eachmel-frame P_(mel)( ), p_(z)(z) represents a Gaussian likelihoodfunction, and J is the Jacobian of an invertible transformation g suchthat z=g⁻¹(x).

In one or more embodiments, inference may be performed as represented byEquation (3).z˜N(O,I) and x=g(z)  (3)

During inference, a phoneme flow may be utilized to attain per-phonemedurations, which may correspond to the alignment A, which are used toprepare the input to a parallel mel-decoder flow that models P_(mel)( ).This decoder may sample latent vectors, such as from the MLE 212, andmap them to plausible-sounding mel-frames, as noted herein. Alignmentbetween text and speech, without dependencies on external aligners, isan important process for providing an end-to-end TTS system. Variousembodiments of the present disclosure may utilize one or more affinecoupling layers to split input data for use to infer scale andtranslation parameters. In various embodiments, inputs may be conditionson context, which may be related to a temporal dimension. The affinecoupling layer may be used to generate one or more context matrices.This may be used along with the speaker embedding vector in order toprovide a matrix with temporal alignment between text information andmel-spectrogram frames, as well as provide speaker-dependentinformation. Recent work may utilize a flow-based model for mappingmel-spectrogram frames to latent vectors. While this method may yieldstable results, non-invertible architectures often lead to attentioncollapsing to trivial solutions, which may limit transferability toother models. Moreover, as noted above, the flow-based model does notaccount for the loss in inference diversity. Accordingly, the systemsand methods of the present disclosure continue to illustrateimprovements over existing models.

An alignment architecture 270 is illustrated in FIG. 2C, which may formone or more portions of a machine learning system that may be utilizedwith embodiments of the present disclosure. In this example, input text210 and mel frames 272, which may correspond, at least in part tomel-spectrogram 202, are both encoded at an encoder 274, which mayinclude separate encoders for the mel frames and the input text 210. Forexample, a text encoder may evaluate text tokens (e.g., phonemeembeddings). In at least one embodiment, each of the input 210 andframes 272 are coded with 1D convnets with a limited receptive field toincorporate local context. The encoder 274 generates an output thatserves an input to a softmax function 276, which may generateprobability matrix 278 that includes one or more vectors ofprobabilities for the text and mel frames, which may be utilized togenerate one or more visualizations, as noted herein.

Various embodiments may enable unsupervised or partially unsupervisedalignment learning. The alignment may be developed without anydependencies on external aligners. In at least one embodiment, acombination Viterbi and forward-backward algorithms used in HiddenMarkov Models (HMMs) may be utilized in order to learn both hard andsoft alignments, respectively. As noted above, alignment may berepresented as A∈

^(N×T), where different alignments may be designated as a “hard”alignment or a “soft” alignment. A_(soft) may represent alignmentbetween text Φ and mel frames X of lengths N and T respectively, suchthat every column of A_(soft) is normalized to a probabilitydistribution. A_(soft) may be processed to extract monotonic, binarizedalignment matrices A_(hard) such that for every frame the probabilitymass is concentrated on a single symbol, and

$\underset{j = 1}{\sum\limits^{T}}A_{{hard},i,j}$yields a vector of durations of every symbol.

Soft alignment may be based on a learned pairwise affinity between alltext tokens ο∈Φ and mel-frames x∈X, which may be normalized with softmaxacross the text dimension represented in Equations (4) and (5).D _(i,j)=dist_(L2)(ϕ_(i) ^(enc) x _(j) ^(enc)),  (4)A _(soft)=softmax(−D,dim=0)  (5)

With respect to Equations (4) and (5), x^(enc) and ϕ^(enc) are encodedvariants of x and ϕ, each using a 2 or 3 1D convolution layer. Invarious embodiments, a loss module may be utilized to develop amonotonic sequence that starts and ends at the first and last texttokens respectively, uses each text token once, and enables advancementby 0 or 1 text tokens for every advance of mel frame.

Various embodiments enable acceleration of alignment learning using aprior that promotes the elements on a near-diagonal path. Abeta-binomial distribution may be used to encourage forward movement ofthe attention mechanism. This beta-binomial distribution may be used toconstruct a 2D cigar-shaped prior over a diagonal of A_(soft), whichwidens in the center of the matrix and narrows towards the corners. Theprior may be weighted or scaled, where a lower weight may increase awidth of the prior.

Because alignments generated through duration predication are inherentlybinary, the model may be conditioned on a binarized alignment matrix toavoid creating a train-test domain gap. This may be achieved using theViterbi algorithm while applying the same constraints for a monotonicalignment, which may provide the most likely monotonic alignment fromthe distribution over monotonic paths. Additionally, results may bemanaged such that A_(soft) matches A_(hard) as much as possible byminimizing loss.

A soft alignment visualization 300 is shown in FIG. 3 , along with abeta-binomial prior visualization 302, a soft alignment with priorvisualization 304, and a hard alignment 306. In these examples, melframes are represented on an x-axis 308 and text tokens are representedon a y-axis 310. The soft alignment 300 illustrates a variety ofdifferent potential alignments across a time period, with the mostlikely alignments shown as lighter shading near the corners (e.g., abottom left corner and a top right corner), corresponding to a start andend time for the audio clip. Application of the prior 312 is shown inthe visualization 302, where the prior 312 has a cigar-shape (e.g., awider middle than edges) and extends substantially along a diagonal fromthe bottom left to the top right. As noted above, this configuration mayenable restriction of alignment at portions likely to be aligned, suchas the beginning (bottom left) and end (top right), thereby potentiallyimproving accuracy. The prior 312 may apply a boundary to limit samplingover a most probable portion of the distribution, as shown invisualization 304, where soft alignment is improved, which may beillustrated by the lighter shading illustrated along the diagonalrepresented by the prior 312. The visualization 306 illustrates an evenmore improved alignment substantially conforming to the angle and sizeof the prior 312. Accordingly, synthesized speech may utilize thealignment developed using the prior in order to determine how to applysampled phoneme durations to generate synthesized speech.

A sample phoneme-level duration distribution 400 is illustrated in FIG.4A. In this example, phonemes forming the world “climate” areillustrated. As shown, the x-axis 402 represents the phonemes 404forming the word “climate,” which in this instance correspond to “k,”“l,” “ayi,” “m,” “and,” and “t.” Each of these phonemes 404A-404Finclude a respective set of distributions 406A-406F corresponding to aduration, represented on the y-axis 408. These distributions may begenerated based upon evaluations of a set of speakers saying the word“climate” to determine how long each phoneme takes to recite. As shown,each of the phonemes 404A-404F may have a slightly different duration,and as a result, attempting to force alignment between text and speechusing a fixed duration will lead to an unnatural sounding audio output.That is, the duration of “ayi” may be approximately twice as long as“k,” and a fixed duration for each phoneme would lead to either a pauseor delay after “k” or a clipping or rushed response of “ayi.” As noted,various embodiments of the present disclosure may enable improveddiversity by sampling over different distributions for various phonemes.By way of example, in this example the phoneme 404A corresponding to “k”has three different distributions 406A (e.g., a set of distributionsthat have slightly different durations). Accordingly, at inference, oneof the distributions may be selected and, within that distribution, aduration may be selected. In another example, using a similar sound, adifferent distribution and duration may be selected. In this manner,improved diversity in synthesized speech is enabled by providingdifferent phoneme durations at inference, or at least, providing alikelihood that a different duration will be selected.

A pitch distribution 420 is illustrated in FIG. 4B, which may be furtherutilized to provide improved diversity with respect to synthetic speech,as described herein. As shown, the x-axis 422 corresponds to time andthe y-axis 424 corresponds to relative pitch. In this example, thedistribution 420 may be computed by evaluating a group of sentences orwords, for example during one or more training stages. During inference,pitch may also be sampled from the distribution 420 to provide furtherdiversity in synthesized speech. That is, different attempts tosynthesize the same word may lead to a diverse set of results, where adifferent pitch is presented. When further combined with the phonemeduration selection of FIG. 4A, the same word may be synthesized whilesounding different, such as from a different speaker altogether.

A synthetic speech system 500 is illustrated in FIG. 5 , which mayinclude one or more components previously described herein. It should beappreciated that components may be grouped for illustrative purposes,but that one or more systems may be integrated into or used withdifferent components of the system 500. Furthermore, one or more systemsmay utilize or otherwise share architecture related to one or moremachine learning systems. Accordingly, different components may bedescribed as a separate module or system based on one or more functions,but may be part of a single integrated system. In this example, a TTSsystem 502 may be utilized to generate synthetic speech based, at leastin part, on a text input 504 and a speaker vector 506. In at least oneembodiment, the speaker vector 506 includes one or more speakerproperties, such as a desired pitch, energy, tone, accent, or the like.In various embodiments, the speaker vector 506 may include weightingproperties corresponding to various parts of speech that may be appliedto the synthetically generated speech. Furthermore, the speaker vector506 may be a tunable component to enable voice conversion, among otherfeatures. Furthermore, in at least one embodiment, the speaker vector506 may be selected from a distribution or from a database of speakervectors. Moreover, in at least one embodiment, different components ofthe speaker vector 506 may be sampled or otherwise obtained from adistribution.

In this example, the input text 504 and speaker vector 506 are providedas input to the TTS system 502, where the input text 504 may beevaluated to determine, at least in part, a duration. For example, aduration module 508 (e.g., duration system) may be used to sample overone or more distributions in order to determine respective durations forindividual phonemes forming the input text 504. As previously discussed,training data may be utilized to generate phoneme distributions andindividual phonemes forming input text 504 may be sampled from thedistributions in order to determine respective lengths. In at least oneembodiment, lengths may be different for each time a phoneme isevaluated, which improves diversity in the output synthesized speech.Determination of the duration may then be utilized during alignment, asdiscussed herein.

Further illustrated are pitch and energy modules 510, 512 that may beutilized to sample respective distributions to obtain pitch (e.g.,fundamental frequency) and energy (e.g., amplitude) for differentphonemes forming the input text 504. In at least one embodiment, atleast a portion of pitch and energy may be determined, at least in part,by the speaker vector 506. For example, the speaker vector 506 mayprovide weights to influence pitch and energy. In certain embodiments,pitch and energy may be determined by speaker vector 506, such as adirected attempt for voice conversion. In various embodiments,distributions for duration, energy, and pitch may be stored in adistribution data store 514, which may be accessed at inference. As willbe appreciated, distributions may vary based on language or alphabets,and moreover, may be updated using one or more training processes. Whileduration, pitch, and energy may be described as components that are usedfor sampling, it should be appreciated that various other data pointsand distributions may be utilized for sampling. By way of example only,various features associated with prosody may be utilized for sampling aswell, such as emphasis, contrast, focus, or one or more elements nototherwise represented in grammatical choices by a speaker. Furthermore,additional factors of evaluation may include intonation, stress, tempo,rhythm, pause, and the like.

In at least one embodiment, alignment between input text and outputaudio may be performed via one or more alignment modules 516, which mayinclude various machine learning systems that enable, in part, recursivecomputations over a set of probabilities in order to determine a mostlikely or highest probability alignment between different phonemesforming the input text 504. The alignment likelihood between a phoneme(from the text) and a mel sample (from the audio training data) is basedon the L2 distance, which can be interpreted as proportional to Gaussianlikelihood, as noted herein. In this example, an audio length module 520may determine audio length based, at least in part, on the input text504 and/or the duration of phonemes forming the input text. Moreover, atext length module 522 may determine a text length, which may then becorrelated to the audio length and presented as a matrix, which may beformed by the matrix generation module 524. In at least one embodiment,the matrix generation module 524 may be utilized to generate a matrix ofpotential phoneme durations with respect to their position and then theprior module 518 may apply the prior to the matrix in order to bound orotherwise restrict evaluation to a most likely position. By way ofexample, a cigar-shaped prior may be constructed from a beta-binomialdistribution, as shown in Equation (6)P(mel,text,alignment)=P(mel_(t)text_(n)|alignment)P(alignment),  (6)

where P(alignment) is a betabinomial cigar shaped prior and P(mel,text|alignment) is the L2 distance between the mel sample at time step tand the nth text phoneme in the sequence. As noted herein, the prior mayinclude boundaries and be positioned at a diagonal stretching from abottom left corner to a top right corner. It should be appreciated thatthe prior may be tunable such that different boundaries may be utilized.Accordingly, alignment between the input text 504 and an audio durationmay be generated.

In at least one embodiment, an audio generation module 526 may generateoutput audio based, at least in part, on the alignment and the inputtext 504, among other features such as the speaker vector 506 and/or thepitch, energy, and the like. In various embodiments, audio generationmay have improved diversity due to the sampling from variousdistributions at inference, which may change or modify different phonemedurations, which may lead to different alignments, and therefore,different output speech. Furthermore, changes in sampling to the pitchor energy may further improve on the output diversity.

Various embodiments may also include one or more training systems 528that may enable generation of synthetic training data for use inimproving the TTS system 502. In various embodiments, the trainingsystem 528 may enable distribution aware augmentation where synthetictraining information may be generated, used in training, and thenremoved at inference. By way of example, a synthetic speech generation530 may obtain samples from a sample data store 532 and modify one ormore properties of the speech, such as changing a pitch or an energy, ormodifying various phoneme lengths, among other possibilities. Thissynthetic speech may be labeled or otherwise identified using a labelmodule 534, which may attach information, such as metadata, to thesynthetic speech for identification at a later time. This identificationmay be applied to all synthetic speech, where synthetic speech may beconsidered “dirty” or otherwise “unclean” when compared to actual groundtruth training data. In other embodiments, one or more properties may beevaluated to determine whether data is considered clean or dirty, suchas identification of one or more features for comparison against athreshold. A distribution generator 536 may then modify or generate oneor more distributions using information from the synthetic speech. Thesegenerated distributions may then be used for training purposes, butduring inference, synthetic speech may be identified, such as using thelabels, and then removed. In this manner, additional training data maybe generated to improve the model, but at inference, only data formingthe ground truth information is used for generation of synthetic speechresponsive to the user input.

FIG. 6A illustrates an example process 600 for generating synthesizedspeech. It should be understood that for this and other processespresented herein that there can be additional, fewer, or alternativesteps performed in similar or alternative order, or at least partiallyin parallel, within the scope of various embodiments unless otherwisespecifically stated. In this example, a plurality of audio segments arereceived 602. The audio segments may form at least part of a set oftraining data that is evaluated to determine different aspects ofdifferent parts of speech. In various embodiments, the audio segmentsmay be subject to one or more pre-processing or processing steps toextract different portions of the audio segment, such as phonemesforming words, pitch, energy, and the like. In at least one embodiment,phoneme durations, phoneme pitches, and phoneme energies are determinedfrom the plurality of audio segments 604. Information extracted from theplurality of phonemes may then be utilized to generate one or moredistributions indicative of certain features found in the audiosegments. For example, phoneme durations may be used to generate a firstdistribution 606, phoneme pitches may be used to generate a seconddistribution 608, and phoneme energies may be used to generate a thirddistribution 610. It should be appreciated that additional distributionsto capture other parts of speech or features of the audio segments mayalso be generated.

In various embodiments, the distributions may be utilized to generatesynthetic speech, such as associated with a conversational AI. A systemmay receive a text input, which is represented as a sequence of text,and determine an alignment between the sequence of text and an audiolength 612. The alignment may be based, at least in part, on the firstdistribution, which correlates different phonemes forming the text inputto their respective durations. In various embodiments, distributions aresampled for respective phonemes in order to select a phoneme duration.It should be appreciated that this type of probabilistic sampling mayimprove diversity in generated synthetic speech because, by sampling atinference, the same text input may be presented as different outputaudio, for example, due to differences in phoneme duration, among otherfactors. Accordingly, synthesized speech may be generated based, atleast in part, on the alignment, the second distribution, and the thirddistribution 614. Moreover, as noted above, in various embodiments oneor more additional distributions may also be utilized to generate thesynthesized speech.

FIG. 6B illustrates an example process 620 for generating synthesizedspeech. In this example, respective alignments between text of aplurality of audio segments and a duration of the plurality of audiosegments is determined 622. For example, a plurality of audio segmentsmay correspond to training information that is determined to determine adistribution of phoneme lengths for different words or phrases. Analignment distribution may be generated, which may be based, at least inpart, on the respective alignments 624. The alignment distribution maybe presented in the form of a matrix that illustrates a probabilisticlikelihood that a certain phoneme will align with a certain portion of atext sample. In at least one embodiment, one or more vectors, from thealignment distribution, are determined 626. The one or more vectors maycorrespond to one or more speaker characteristics.

As noted, various embodiments may be used to generate synthesizedspeech, where a text sequence is received 628. The text sequence may beinput by a user or extracted from an image, among other options. Asynthetic audio clip may be generated based, at least in part, on thetext sequence and the one or more vectors, where the synthetic audioclip corresponds to the text sequence 630.

FIG. 6C illustrates an example process 650 for training a TTS system. Inthis example, one or more synthetic training clips are generated 652.The synthetic training clips may be generated based, at least in part,on one or more sample audio segments, which may correspond to groundtruth training data provided for training the TTS system, among otheroptions. In at least one embodiment, synthetic training clips modify oneor more properties of the sample audio segments, such as changing apitch or speed of the speech. The synthetic training clips may belabeled 654, such as by identifying their locations within adistribution or by associating metadata with the clips. One or moremachine learning systems may then be trained using at least some of thesample audio segments and at least some of the one or more synthetictraining clips 656. The synthetic training clips may enable a largertraining set, which may improve later inferences.

In at least one embodiment, the trained machine learning system isutilized to generate synthetic audio clips. A request may be received togenerate synthetic speech 658. This speech may be generated byprocessing a text input using the TTS system, which may align parts ofspeech with different durations in order to generate output audio. In atleast one embodiment, one or more locations within a distributionassociated with synthetic training clips are identified 660, and theselocations are avoided or not sampled from during generation of thesynthetic speech 662. In this manner, synthetic training clips may beused to improve modeling by increasing a data set, but may not be usedduring inferencing.

Data Center

FIG. 7 illustrates an example data center 700, in which at least oneembodiment may be used. In at least one embodiment, data center 700includes a data center infrastructure layer 710, a framework layer 720,a software layer 730, and an application layer 740.

In at least one embodiment, as shown in FIG. 7 , data centerinfrastructure layer 710 may include a resource orchestrator 712,grouped computing resources 714, and node computing resources (“nodeC.R.s”) 716(1)-716(N), where “N” represents any whole, positive integer.In at least one embodiment, node C.R.s 716(1)-716(N) may include, butare not limited to, any number of central processing units (“CPUs”) orother processors (including accelerators, field programmable gate arrays(FPGAs), graphics processors, etc.), memory devices (e.g., dynamicread-only memory), storage devices (e.g., solid state or disk drives),network input/output (“NW I/O”) devices, network switches, virtualmachines (“VMs”), power modules, and cooling modules, etc. In at leastone embodiment, one or more node C.R.s from among node C.R.s716(1)-716(N) may be a server having one or more of above-mentionedcomputing resources.

In at least one embodiment, grouped computing resources 714 may includeseparate groupings of node C.R.s housed within one or more racks (notshown), or many racks housed in data centers at various geographicallocations (also not shown). Separate groupings of node C.R.s withingrouped computing resources 714 may include grouped compute, network,memory or storage resources that may be configured or allocated tosupport one or more workloads. In at least one embodiment, several nodeC.R.s including CPUs or processors may grouped within one or more racksto provide compute resources to support one or more workloads. In atleast one embodiment, one or more racks may also include any number ofpower modules, cooling modules, and network switches, in anycombination.

In at least one embodiment, resource orchestrator 712 may configure orotherwise control one or more node C.R.s 716(1)-716(N) and/or groupedcomputing resources 714. In at least one embodiment, resourceorchestrator 712 may include a software design infrastructure (“SDI”)management entity for data center 700. In at least one embodiment,resource orchestrator may include hardware, software or some combinationthereof.

In at least one embodiment, as shown in FIG. 7 , framework layer 720includes a job scheduler 722, a configuration manager 724, a resourcemanager 726 and a distributed file system 728. In at least oneembodiment, framework layer 720 may include a framework to supportsoftware 732 of software layer 730 and/or one or more application(s) 742of application layer 740. In at least one embodiment, software 732 orapplication(s) 742 may respectively include web-based service softwareor applications, such as those provided by Amazon Web Services, GoogleCloud and Microsoft Azure. In at least one embodiment, framework layer720 may be, but is not limited to, a type of free and open-sourcesoftware web application framework such as Apache Spark™ (hereinafter“Spark”) that may utilize distributed file system 728 for large-scaledata processing (e.g., “big data”). In at least one embodiment, jobscheduler 722 may include a Spark driver to facilitate scheduling ofworkloads supported by various layers of data center 700. In at leastone embodiment, configuration manager 724 may be capable of configuringdifferent layers such as software layer 730 and framework layer 720including Spark and distributed file system 728 for supportinglarge-scale data processing. In at least one embodiment, resourcemanager 726 may be capable of managing clustered or grouped computingresources mapped to or allocated for support of distributed file system728 and job scheduler 722. In at least one embodiment, clustered orgrouped computing resources may include grouped computing resource 714at data center infrastructure layer 710. In at least one embodiment,resource manager 726 may coordinate with resource orchestrator 712 tomanage these mapped or allocated computing resources.

In at least one embodiment, software 732 included in software layer 730may include software used by at least portions of node C.R.s716(1)-716(N), grouped computing resources 714, and/or distributed filesystem 728 of framework layer 720. The one or more types of software mayinclude, but are not limited to, Internet web page search software,e-mail virus scan software, database software, and streaming videocontent software.

In at least one embodiment, application(s) 742 included in applicationlayer 740 may include one or more types of applications used by at leastportions of node C.R.s 716(1)-716(N), grouped computing resources 714,and/or distributed file system 728 of framework layer 720. One or moretypes of applications may include, but are not limited to, any number ofa genomics application, a cognitive compute, and a machine learningapplication, including training or inferencing software, machinelearning framework software (e.g., PyTorch, TensorFlow, Caffe, etc.) orother machine learning applications used in conjunction with one or moreembodiments.

In at least one embodiment, any of configuration manager 724, resourcemanager 726, and resource orchestrator 712 may implement any number andtype of self-modifying actions based on any amount and type of dataacquired in any technically feasible fashion. In at least oneembodiment, self-modifying actions may relieve a data center operator ofdata center 700 from making possibly bad configuration decisions andpossibly avoiding underutilized and/or poor performing portions of adata center.

In at least one embodiment, data center 700 may include tools, services,software or other resources to train one or more machine learning modelsor predict or infer information using one or more machine learningmodels according to one or more embodiments described herein. Forexample, in at least one embodiment, a machine learning model may betrained by calculating weight parameters according to a neural networkarchitecture using software and computing resources described above withrespect to data center 700. In at least one embodiment, trained machinelearning models corresponding to one or more neural networks may be usedto infer or predict information using resources described above withrespect to data center 700 by using weight parameters calculated throughone or more training techniques described herein.

In at least one embodiment, data center may use CPUs,application-specific integrated circuits (ASICs), GPUs, FPGAs, or otherhardware to perform training and/or inferencing using above-describedresources. Moreover, one or more software and/or hardware resourcesdescribed above may be configured as a service to allow users to trainor performing inferencing of information, such as image recognition,speech recognition, or other artificial intelligence services.

Such components can be used for generating synthetic speech.

Computer Systems

FIG. 8 is a block diagram illustrating an exemplary computer system,which may be a system with interconnected devices and components, asystem-on-a-chip (SOC) or some combination thereof 800 formed with aprocessor that may include execution units to execute an instruction,according to at least one embodiment. In at least one embodiment,computer system 800 may include, without limitation, a component, suchas a processor 802 to employ execution units including logic to performalgorithms for process data, in accordance with present disclosure, suchas in embodiment described herein. In at least one embodiment, computersystem 800 may include processors, such as PENTIUM® Processor family,Xeon™, Itanium®, XScale™ and/or StrongARM™, Intel® Core™, or Intel®Nervana™ microprocessors available from Intel Corporation of SantaClara, Calif., although other systems (including PCs having othermicroprocessors, engineering workstations, set-top boxes and like) mayalso be used. In at least one embodiment, computer system 800 mayexecute a version of WINDOWS' operating system available from MicrosoftCorporation of Redmond, Wash., although other operating systems (UNIXand Linux for example), embedded software, and/or graphical userinterfaces, may also be used.

Embodiments may be used in other devices such as handheld devices andembedded applications. Some examples of handheld devices includecellular phones, Internet Protocol devices, digital cameras, personaldigital assistants (“PDAs”), and handheld PCs. In at least oneembodiment, embedded applications may include a microcontroller, adigital signal processor (“DSP”), system on a chip, network computers(“NetPCs”), edge computing devices, set-top boxes, network hubs, widearea network (“WAN”) switches, or any other system that may perform oneor more instructions in accordance with at least one embodiment.

In at least one embodiment, computer system 800 may include, withoutlimitation, processor 802 that may include, without limitation, one ormore execution units 808 to perform machine learning model trainingand/or inferencing according to techniques described herein. In at leastone embodiment, computer system 800 is a single processor desktop orserver system, but in another embodiment computer system 800 may be amultiprocessor system. In at least one embodiment, processor 802 mayinclude, without limitation, a complex instruction set computer (“CISC”)microprocessor, a reduced instruction set computing (“RISC”)microprocessor, a very long instruction word (“VLIW”) microprocessor, aprocessor implementing a combination of instruction sets, or any otherprocessor device, such as a digital signal processor, for example. In atleast one embodiment, processor 802 may be coupled to a processor bus810 that may transmit data signals between processor 802 and othercomponents in computer system 800.

In at least one embodiment, processor 802 may include, withoutlimitation, a Level 1 (“L1”) internal cache memory (“cache”) 804. In atleast one embodiment, processor 802 may have a single internal cache ormultiple levels of internal cache. In at least one embodiment, cachememory may reside external to processor 802. Other embodiments may alsoinclude a combination of both internal and external caches depending onparticular implementation and needs. In at least one embodiment,register file 806 may store different types of data in various registersincluding, without limitation, integer registers, floating pointregisters, status registers, and instruction pointer register.

In at least one embodiment, execution unit 808, including, withoutlimitation, logic to perform integer and floating point operations, alsoresides in processor 802. In at least one embodiment, processor 802 mayalso include a microcode (“ucode”) read only memory (“ROM”) that storesmicrocode for certain macro instructions. In at least one embodiment,execution unit 808 may include logic to handle a packed instruction set809. In at least one embodiment, by including packed instruction set 809in an instruction set of a general-purpose processor 802, along withassociated circuitry to execute instructions, operations used by manymultimedia applications may be performed using packed data in ageneral-purpose processor 802. In one or more embodiments, manymultimedia applications may be accelerated and executed more efficientlyby using full width of a processor's data bus for performing operationson packed data, which may eliminate need to transfer smaller units ofdata across processor's data bus to perform one or more operations onedata element at a time.

In at least one embodiment, execution unit 808 may also be used inmicrocontrollers, embedded processors, graphics devices, DSPs, and othertypes of logic circuits. In at least one embodiment, computer system 800may include, without limitation, a memory 820. In at least oneembodiment, memory 820 may be implemented as a Dynamic Random AccessMemory (“DRAM”) device, a Static Random Access Memory (“SRAM”) device,flash memory device, or other memory device. In at least one embodiment,memory 820 may store instruction(s) 819 and/or data 821 represented bydata signals that may be executed by processor 802.

In at least one embodiment, system logic chip may be coupled toprocessor bus 810 and memory 820. In at least one embodiment, systemlogic chip may include, without limitation, a memory controller hub(“MCH”) 816, and processor 802 may communicate with MCH 816 viaprocessor bus 810. In at least one embodiment, MCH 816 may provide ahigh bandwidth memory path 818 to memory 820 for instruction and datastorage and for storage of graphics commands, data and textures. In atleast one embodiment, MCH 816 may direct data signals between processor802, memory 820, and other components in computer system 800 and tobridge data signals between processor bus 810, memory 820, and a systemI/O 822. In at least one embodiment, system logic chip may provide agraphics port for coupling to a graphics controller. In at least oneembodiment, MCH 816 may be coupled to memory 820 through a highbandwidth memory path 818 and graphics/video card 812 may be coupled toMCH 816 through an Accelerated Graphics Port (“AGP”) interconnect 814.

In at least one embodiment, computer system 800 may use system I/O 822that is a proprietary hub interface bus to couple MCH 816 to I/Ocontroller hub (“ICH”) 830. In at least one embodiment, ICH 830 mayprovide direct connections to some I/O devices via a local I/O bus. Inat least one embodiment, local I/O bus may include, without limitation,a high-speed I/O bus for connecting peripherals to memory 820, chipset,and processor 802. Examples may include, without limitation, an audiocontroller 829, a firmware hub (“flash BIOS”) 828, a wirelesstransceiver 826, a data storage 824, a legacy I/O controller 823containing user input and keyboard interfaces 825, a serial expansionport 827, such as Universal Serial Bus (“USB”), and a network controller834. Data storage 824 may comprise a hard disk drive, a floppy diskdrive, a CD-ROM device, a flash memory device, or other mass storagedevice.

In at least one embodiment, FIG. 8 illustrates a system, which includesinterconnected hardware devices or “chips”, whereas in otherembodiments, FIG. 8 may illustrate an exemplary System on a Chip(“SoC”). In at least one embodiment, devices may be interconnected withproprietary interconnects, standardized interconnects (e.g., PCIe) orsome combination thereof. In at least one embodiment, one or morecomponents of computer system 800 are interconnected using computeexpress link (CXL) interconnects.

Such components can be used for generating synthetic speech.

FIG. 9 is a block diagram illustrating an electronic device 900 forutilizing a processor 910, according to at least one embodiment. In atleast one embodiment, electronic device 900 may be, for example andwithout limitation, a notebook, a tower server, a rack server, a bladeserver, a laptop, a desktop, a tablet, a mobile device, a phone, anembedded computer, or any other suitable electronic device.

In at least one embodiment, system 900 may include, without limitation,processor 910 communicatively coupled to any suitable number or kind ofcomponents, peripherals, modules, or devices. In at least oneembodiment, processor 910 coupled using a bus or interface, such as a 1°C. bus, a System Management Bus (“SMBus”), a Low Pin Count (LPC) bus, aSerial Peripheral Interface (“SPI”), a High Definition Audio (“HDA”)bus, a Serial Advance Technology Attachment (“SATA”) bus, a UniversalSerial Bus (“USB”) (versions 1, 2, 3), or a Universal AsynchronousReceiver/Transmitter (“UART”) bus. In at least one embodiment, FIG. 9illustrates a system, which includes interconnected hardware devices or“chips”, whereas in other embodiments, FIG. 9 may illustrate anexemplary System on a Chip (“SoC”). In at least one embodiment, devicesillustrated in FIG. 9 may be interconnected with proprietaryinterconnects, standardized interconnects (e.g., PCIe) or somecombination thereof. In at least one embodiment, one or more componentsof FIG. 9 are interconnected using compute express link (CXL)interconnects.

In at least one embodiment, FIG. 9 may include a display 924, a touchscreen 925, a touch pad 930, a Near Field Communications unit (“NFC”)945, a sensor hub 940, a thermal sensor 946, an Express Chipset (“EC”)935, a Trusted Platform Module (“TPM”) 938, BIOS/firmware/flash memory(“BIOS, FW Flash”) 922, a DSP 960, a drive 920 such as a Solid StateDisk (“SSD”) or a Hard Disk Drive (“HDD”), a wireless local area networkunit (“WLAN”) 950, a Bluetooth unit 952, a Wireless Wide Area Networkunit (“WWAN”) 956, a Global Positioning System (GPS) 955, a camera (“USB3.0 camera”) 954 such as a USB 3.0 camera, and/or a Low Power DoubleData Rate (“LPDDR”) memory unit (“LPDDR3”) 915 implemented in, forexample, LPDDR3 standard. These components may each be implemented inany suitable manner.

In at least one embodiment, other components may be communicativelycoupled to processor 910 through components discussed above. In at leastone embodiment, an accelerometer 941, Ambient Light Sensor (“ALS”) 942,compass 943, and a gyroscope 944 may be communicatively coupled tosensor hub 940. In at least one embodiment, thermal sensor 939, a fan937, a keyboard 946, and a touch pad 930 may be communicatively coupledto EC 935. In at least one embodiment, speaker 963, headphones 964, andmicrophone (“mic”) 965 may be communicatively coupled to an audio unit(“audio codec and class d amp”) 962, which may in turn becommunicatively coupled to DSP 960. In at least one embodiment, audiounit 964 may include, for example and without limitation, an audiocoder/decoder (“codec”) and a class D amplifier. In at least oneembodiment, SIM card (“SIM”) 957 may be communicatively coupled to WWANunit 956. In at least one embodiment, components such as WLAN unit 950and Bluetooth unit 952, as well as WWAN unit 956 may be implemented in aNext Generation Form Factor (“NGFF”).

Such components can be used for generating synthetic speech.

FIG. 10 is a block diagram of a processing system, according to at leastone embodiment. In at least one embodiment, system 1000 includes one ormore processors 1002 and one or more graphics processors 1008, and maybe a single processor desktop system, a multiprocessor workstationsystem, or a server system or datacenter having a large number ofcollectively or separably managed processors 1002 or processor cores1007. In at least one embodiment, system 1000 is a processing platformincorporated within a system-on-a-chip (SoC) integrated circuit for usein mobile, handheld, or embedded devices.

In at least one embodiment, system 1000 can include, or be incorporatedwithin a server-based gaming platform, a cloud computing host platform,a virtualized computing platform, a game console, including a game andmedia console, a mobile gaming console, a handheld game console, or anonline game console. In at least one embodiment, system 1000 is a mobilephone, smart phone, tablet computing device or mobile Internet device.In at least one embodiment, processing system 1000 can also include,couple with, or be integrated within a wearable device, such as a smartwatch wearable device, smart eyewear device, augmented reality device,edge device, Internet of Things (“IoT”) device, or virtual realitydevice. In at least one embodiment, processing system 1000 is atelevision or set top box device having one or more processors 1002 anda graphical interface generated by one or more graphics processors 1008.

In at least one embodiment, one or more processors 1002 each include oneor more processor cores 1007 to process instructions which, whenexecuted, perform operations for system and user software. In at leastone embodiment, each of one or more processor cores 1007 is configuredto process a specific instruction set 1009. In at least one embodiment,instruction set 1009 may facilitate Complex Instruction Set Computing(CISC), Reduced Instruction Set Computing (RISC), or computing via aVery Long Instruction Word (VLIW). In at least one embodiment, processorcores 1007 may each process a different instruction set 1009, which mayinclude instructions to facilitate emulation of other instruction sets.In at least one embodiment, processor core 1007 may also include otherprocessing devices, such a Digital Signal Processor (DSP).

In at least one embodiment, processor 1002 includes cache memory 1004.In at least one embodiment, processor 1002 can have a single internalcache or multiple levels of internal cache. In at least one embodiment,cache memory is shared among various components of processor 1002. In atleast one embodiment, processor 1002 also uses an external cache (e.g.,a Level-3 (L3) cache or Last Level Cache (LLC)) (not shown), which maybe shared among processor cores 1007 using known cache coherencytechniques. In at least one embodiment, register file 1006 isadditionally included in processor 1002 which may include differenttypes of registers for storing different types of data (e.g., integerregisters, floating point registers, status registers, and aninstruction pointer register). In at least one embodiment, register file1006 may include general-purpose registers or other registers.

In at least one embodiment, one or more processor(s) 1002 are coupledwith one or more interface bus(es) 1010 to transmit communicationsignals such as address, data, or control signals between processor 1002and other components in system 1000. In at least one embodiment,interface bus 1010, in one embodiment, can be a processor bus, such as aversion of a Direct Media Interface (DMI) bus. In at least oneembodiment, interface 1010 is not limited to a DMI bus, and may includeone or more Peripheral Component Interconnect buses (e.g., PCI, PCIExpress), memory busses, or other types of interface busses. In at leastone embodiment processor(s) 1002 include an integrated memory controller1016 and a platform controller hub 1030. In at least one embodiment,memory controller 1016 facilitates communication between a memory deviceand other components of system 1000, while platform controller hub (PCH)1030 provides connections to I/O devices via a local I/O bus.

In at least one embodiment, memory device 1020 can be a dynamic randomaccess memory (DRAM) device, a static random access memory (SRAM)device, flash memory device, phase-change memory device, or some othermemory device having suitable performance to serve as process memory. Inat least one embodiment memory device 1020 can operate as system memoryfor system 1000, to store data 1022 and instructions 1021 for use whenone or more processors 1002 executes an application or process. In atleast one embodiment, memory controller 1016 also couples with anoptional external graphics processor 1012, which may communicate withone or more graphics processors 1008 in processors 1002 to performgraphics and media operations. In at least one embodiment, a displaydevice 1011 can connect to processor(s) 1002. In at least one embodimentdisplay device 1011 can include one or more of an internal displaydevice, as in a mobile electronic device or a laptop device or anexternal display device attached via a display interface (e.g.,DisplayPort, etc.). In at least one embodiment, display device 1011 caninclude a head mounted display (HMD) such as a stereoscopic displaydevice for use in virtual reality (VR) applications or augmented reality(AR) applications.

In at least one embodiment, platform controller hub 1030 enablesperipherals to connect to memory device 1020 and processor 1002 via ahigh-speed I/O bus. In at least one embodiment, I/O peripherals include,but are not limited to, an audio controller 1046, a network controller1034, a firmware interface 1028, a wireless transceiver 1026, touchsensors 1025, a data storage device 1024 (e.g., hard disk drive, flashmemory, etc.). In at least one embodiment, data storage device 1024 canconnect via a storage interface (e.g., SATA) or via a peripheral bus,such as a Peripheral Component Interconnect bus (e.g., PCI, PCIExpress). In at least one embodiment, touch sensors 1025 can includetouch screen sensors, pressure sensors, or fingerprint sensors. In atleast one embodiment, wireless transceiver 1026 can be a Wi-Fitransceiver, a Bluetooth transceiver, or a mobile network transceiversuch as a 3G, 4G, or Long Term Evolution (LTE) transceiver. In at leastone embodiment, firmware interface 1028 enables communication withsystem firmware, and can be, for example, a unified extensible firmwareinterface (UEFI). In at least one embodiment, network controller 1034can enable a network connection to a wired network. In at least oneembodiment, a high-performance network controller (not shown) coupleswith interface bus 1010. In at least one embodiment, audio controller1046 is a multi-channel high definition audio controller. In at leastone embodiment, system 1000 includes an optional legacy I/O controller1040 for coupling legacy (e.g., Personal System 2 (PS/2)) devices tosystem. In at least one embodiment, platform controller hub 1030 canalso connect to one or more Universal Serial Bus (USB) controllers 1042connect input devices, such as keyboard and mouse 1043 combinations, acamera 1044, or other USB input devices.

In at least one embodiment, an instance of memory controller 1016 andplatform controller hub 1030 may be integrated into a discreet externalgraphics processor, such as external graphics processor 1012. In atleast one embodiment, platform controller hub 1030 and/or memorycontroller 1016 may be external to one or more processor(s) 1002. Forexample, in at least one embodiment, system 1000 can include an externalmemory controller 1016 and platform controller hub 1030, which may beconfigured as a memory controller hub and peripheral controller hubwithin a system chipset that is in communication with processor(s) 1002.

Such components can be used for generating synthetic speech.

FIG. 11 is a block diagram of a processor 1100 having one or moreprocessor cores 1102A-1102N, an integrated memory controller 1114, andan integrated graphics processor 1108, according to at least oneembodiment. In at least one embodiment, processor 1100 can includeadditional cores up to and including additional core 1102N representedby dashed lined boxes. In at least one embodiment, each of processorcores 1102A-1102N includes one or more internal cache units 1104A-1104N.In at least one embodiment, each processor core also has access to oneor more shared cached units 1106.

In at least one embodiment, internal cache units 1104A-1104N and sharedcache units 1106 represent a cache memory hierarchy within processor1100. In at least one embodiment, cache memory units 1104A-1104N mayinclude at least one level of instruction and data cache within eachprocessor core and one or more levels of shared mid-level cache, such asa Level 2 (L2), Level 3 (L3), Level 4 (L4), or other levels of cache,where a highest level of cache before external memory is classified asan LLC. In at least one embodiment, cache coherency logic maintainscoherency between various cache units 1106 and 1104A-1104N.

In at least one embodiment, processor 1100 may also include a set of oneor more bus controller units 1116 and a system agent core 1110. In atleast one embodiment, one or more bus controller units 1116 manage a setof peripheral buses, such as one or more PCI or PCI express busses. Inat least one embodiment, system agent core 1110 provides managementfunctionality for various processor components. In at least oneembodiment, system agent core 1110 includes one or more integratedmemory controllers 1114 to manage access to various external memorydevices (not shown).

In at least one embodiment, one or more of processor cores 1102A-1102Ninclude support for simultaneous multi-threading. In at least oneembodiment, system agent core 1110 includes components for coordinatingand operating cores 1102A-1102N during multi-threaded processing. In atleast one embodiment, system agent core 1110 may additionally include apower control unit (PCU), which includes logic and components toregulate one or more power states of processor cores 1102A-1102N andgraphics processor 1108.

In at least one embodiment, processor 1100 additionally includesgraphics processor 1108 to execute graphics processing operations. In atleast one embodiment, graphics processor 1108 couples with shared cacheunits 1106, and system agent core 1110, including one or more integratedmemory controllers 1114. In at least one embodiment, system agent core1110 also includes a display controller 1111 to drive graphics processoroutput to one or more coupled displays. In at least one embodiment,display controller 1111 may also be a separate module coupled withgraphics processor 1108 via at least one interconnect, or may beintegrated within graphics processor 1108.

In at least one embodiment, a ring based interconnect unit 1112 is usedto couple internal components of processor 1100. In at least oneembodiment, an alternative interconnect unit may be used, such as apoint-to-point interconnect, a switched interconnect, or othertechniques. In at least one embodiment, graphics processor 1108 coupleswith ring interconnect 1112 via an I/O link 1113.

In at least one embodiment, I/O link 1113 represents at least one ofmultiple varieties of I/O interconnects, including an on package I/Ointerconnect which facilitates communication between various processorcomponents and a high-performance embedded memory module 1118, such asan eDRAM module. In at least one embodiment, each of processor cores1102A-1102N and graphics processor 1108 use embedded memory modules 1118as a shared Last Level Cache.

In at least one embodiment, processor cores 1102A-1102N are homogenouscores executing a common instruction set architecture. In at least oneembodiment, processor cores 1102A-1102N are heterogeneous in terms ofinstruction set architecture (ISA), where one or more of processor cores1102A-1102N execute a common instruction set, while one or more othercores of processor cores 1102A-1102N executes a subset of a commoninstruction set or a different instruction set. In at least oneembodiment, processor cores 1102A-1102N are heterogeneous in terms ofmicroarchitecture, where one or more cores having a relatively higherpower consumption couple with one or more power cores having a lowerpower consumption. In at least one embodiment, processor 1100 can beimplemented on one or more chips or as an SoC integrated circuit.

Such components can be used for generating synthetic speech.

Other variations are within spirit of present disclosure. Thus, whiledisclosed techniques are susceptible to various modifications andalternative constructions, certain illustrated embodiments thereof areshown in drawings and have been described above in detail. It should beunderstood, however, that there is no intention to limit disclosure tospecific form or forms disclosed, but on contrary, intention is to coverall modifications, alternative constructions, and equivalents fallingwithin spirit and scope of disclosure, as defined in appended claims.

Use of terms “a” and “an” and “the” and similar referents in context ofdescribing disclosed embodiments (especially in context of followingclaims) are to be construed to cover both singular and plural, unlessotherwise indicated herein or clearly contradicted by context, and notas a definition of a term. Terms “comprising,” “having,” “including,”and “containing” are to be construed as open-ended terms (meaning“including, but not limited to,”) unless otherwise noted. Term“connected,” when unmodified and referring to physical connections, isto be construed as partly or wholly contained within, attached to, orjoined together, even if there is something intervening. Recitation ofranges of values herein are merely intended to serve as a shorthandmethod of referring individually to each separate value falling withinrange, unless otherwise indicated herein and each separate value isincorporated into specification as if it were individually recitedherein. Use of term “set” (e.g., “a set of items”) or “subset,” unlessotherwise noted or contradicted by context, is to be construed as anonempty collection comprising one or more members. Further, unlessotherwise noted or contradicted by context, term “subset” of acorresponding set does not necessarily denote a proper subset ofcorresponding set, but subset and corresponding set may be equal.

Conjunctive language, such as phrases of form “at least one of A, B, andC,” or “at least one of A, B and C,” unless specifically statedotherwise or otherwise clearly contradicted by context, is otherwiseunderstood with context as used in general to present that an item,term, etc., may be either A or B or C, or any nonempty subset of set ofA and B and C. For instance, in illustrative example of a set havingthree members, conjunctive phrases “at least one of A, B, and C” and “atleast one of A, B and C” refer to any of following sets: {A}, {B}, {C},{A, B}, {A, C}, {B, C}, {A, B, C}. Thus, such conjunctive language isnot generally intended to imply that certain embodiments require atleast one of A, at least one of B, and at least one of C each to bepresent. In addition, unless otherwise noted or contradicted by context,term “plurality” indicates a state of being plural (e.g., “a pluralityof items” indicates multiple items). A plurality is at least two items,but can be more when so indicated either explicitly or by context.Further, unless stated otherwise or otherwise clear from context, phrase“based on” means “based at least in part on” and not “based solely on.”

Operations of processes described herein can be performed in anysuitable order unless otherwise indicated herein or otherwise clearlycontradicted by context. In at least one embodiment, a process such asthose processes described herein (or variations and/or combinationsthereof) is performed under control of one or more computer systemsconfigured with executable instructions and is implemented as code(e.g., executable instructions, one or more computer programs or one ormore applications) executing collectively on one or more processors, byhardware or combinations thereof. In at least one embodiment, code isstored on a computer-readable storage medium, for example, in form of acomputer program comprising a plurality of instructions executable byone or more processors. In at least one embodiment, a computer-readablestorage medium is a non-transitory computer-readable storage medium thatexcludes transitory signals (e.g., a propagating transient electric orelectromagnetic transmission) but includes non-transitory data storagecircuitry (e.g., buffers, cache, and queues) within transceivers oftransitory signals. In at least one embodiment, code (e.g., executablecode or source code) is stored on a set of one or more non-transitorycomputer-readable storage media having stored thereon executableinstructions (or other memory to store executable instructions) that,when executed (i.e., as a result of being executed) by one or moreprocessors of a computer system, cause computer system to performoperations described herein. A set of non-transitory computer-readablestorage media, in at least one embodiment, comprises multiplenon-transitory computer-readable storage media and one or more ofindividual non-transitory storage media of multiple non-transitorycomputer-readable storage media lack all of code while multiplenon-transitory computer-readable storage media collectively store all ofcode. In at least one embodiment, executable instructions are executedsuch that different instructions are executed by differentprocessors—for example, a non-transitory computer-readable storagemedium store instructions and a main central processing unit (“CPU”)executes some of instructions while a graphics processing unit (“GPU”)and/or a data processing unit (“DPU”) executes other instructions. In atleast one embodiment, different components of a computer system haveseparate processors and different processors execute different subsetsof instructions.

Accordingly, in at least one embodiment, computer systems are configuredto implement one or more services that singly or collectively performoperations of processes described herein and such computer systems areconfigured with applicable hardware and/or software that enableperformance of operations. Further, a computer system that implements atleast one embodiment of present disclosure is a single device and, inanother embodiment, is a distributed computer system comprising multipledevices that operate differently such that distributed computer systemperforms operations described herein and such that a single device doesnot perform all operations.

Use of any and all examples, or exemplary language (e.g., “such as”)provided herein, is intended merely to better illuminate embodiments ofdisclosure and does not pose a limitation on scope of disclosure unlessotherwise claimed. No language in specification should be construed asindicating any non-claimed element as essential to practice ofdisclosure.

All references, including publications, patent applications, andpatents, cited herein are hereby incorporated by reference to sameextent as if each reference were individually and specifically indicatedto be incorporated by reference and were set forth in its entiretyherein.

In description and claims, terms “coupled” and “connected,” along withtheir derivatives, may be used. It should be understood that these termsmay be not intended as synonyms for each other. Rather, in particularexamples, “connected” or “coupled” may be used to indicate that two ormore elements are in direct or indirect physical or electrical contactwith each other. “Coupled” may also mean that two or more elements arenot in direct contact with each other, but yet still co-operate orinteract with each other.

Unless specifically stated otherwise, it may be appreciated thatthroughout specification terms such as “processing,” “computing,”“calculating,” “determining,” or like, refer to action and/or processesof a computer or computing system, or similar electronic computingdevice, that manipulate and/or transform data represented as physical,such as electronic, quantities within computing system's registersand/or memories into other data similarly represented as physicalquantities within computing system's memories, registers or other suchinformation storage, transmission or display devices.

In a similar manner, term “processor” may refer to any device or portionof a device that processes electronic data from registers and/or memoryand transform that electronic data into other electronic data that maybe stored in registers and/or memory. As non-limiting examples,“processor” may be any processor capable of general purpose processingsuch as a CPU, GPU, or DPU. As non-limiting examples, “processor” may beany microcontroller or dedicated processing unit such as a DSP, imagesignal processor (“ISP”), arithmetic logic unit (“ALU”), visionprocessing unit (“VPU”), tree traversal unit (“TTU”), ray tracing core,tensor tracing core, tensor processing unit (“TPU”), embedded controlunit (“ECU”), and the like. As non-limiting examples, “processor” may bea hardware accelerator, such as a PVA (programmable vision accelerator),DLA (deep learning accelerator), etc. As non-limiting examples,“processor” may also include one or more virtual instances of a CPU,GPU, etc., hosted on an underlying hardware component executing one ormore virtual machines. A “computing platform” may comprise one or moreprocessors. As used herein, “software” processes may include, forexample, software and/or hardware entities that perform work over time,such as tasks, threads, and intelligent agents. Also, each process mayrefer to multiple processes, for carrying out instructions in sequenceor in parallel, continuously or intermittently. Terms “system” and“method” are used herein interchangeably insofar as system may embodyone or more methods and methods may be considered a system.

In present document, references may be made to obtaining, acquiring,receiving, or inputting analog or digital data into a subsystem,computer system, or computer-implemented machine. Obtaining, acquiring,receiving, or inputting analog and digital data can be accomplished in avariety of ways such as by receiving data as a parameter of a functioncall or a call to an application programming interface. In someimplementations, process of obtaining, acquiring, receiving, orinputting analog or digital data can be accomplished by transferringdata via a serial or parallel interface. In another implementation,process of obtaining, acquiring, receiving, or inputting analog ordigital data can be accomplished by transferring data via a computernetwork from providing entity to acquiring entity. References may alsobe made to providing, outputting, transmitting, sending, or presentinganalog or digital data. In various examples, process of providing,outputting, transmitting, sending, or presenting analog or digital datacan be accomplished by transferring data as an input or output parameterof a function call, a parameter of an application programming interfaceor interprocess communication mechanism.

Although discussion above sets forth example implementations ofdescribed techniques, other architectures may be used to implementdescribed functionality, and are intended to be within scope of thisdisclosure. Furthermore, although specific distributions ofresponsibilities are defined above for purposes of discussion, variousfunctions and responsibilities might be distributed and divided indifferent ways, depending on circumstances.

Furthermore, although subject matter has been described in languagespecific to structural features and/or methodological acts, it is to beunderstood that subject matter claimed in appended claims is notnecessarily limited to specific features or acts described. Rather,specific features and acts are disclosed as exemplary forms ofimplementing the claims.

What is claimed is:
 1. A computer-implemented method, comprising:determining, from a plurality of audio segments, respective phonemedurations, phoneme pitches, and phoneme energies; determining a firstalignment between a sequence of text and a total speech durationcorresponding to probable locations for the respective phonemedurations; determining a second alignment for an audio segment ofsynthesized speech based, at least in part, on a first distributioncorresponding to the phoneme durations and the first alignment; andgenerating, for the sequence of text, an audio segment comprising asynthesized recitation of the sequence of text based, at least in part,on the second alignment and at least one of a second distributioncorresponding to the phoneme pitches, or a third distributioncorresponding to the phoneme energies.
 2. The computer-implementedmethod of claim 1, further comprising: generating a fourth distributioncorresponding to one or more properties associated with the synthesizedrecitation based, at least in part, on the fourth distribution.
 3. Thecomputer-implemented method of claim 1, further comprising: applying, tothe second alignment, a prior distribution to exclude pairs of phonemesand durations from the plurality of audio segments that are outside of aspecified range.
 4. The computer-implemented method of claim 3, whereinthe prior distribution is cigar-shaped.
 5. The computer-implementedmethod of claim 3, wherein the prior distribution is constructed from abeta-binomial distribution.
 6. The computer-implemented method of claim1, further comprising: determining, from the sequence of text, aplurality of text tokens; and aligning each of the plurality of texttokens to a respective mel frame based, at least in part, on the secondalignment.
 7. The computer-implemented method of claim 6, wherein thesecond alignment is based, at least in part, on an L2 distance betweenthe mel frame at a first time and a text phoneme in the sequence oftext.
 8. The computer-implemented method of claim 1, wherein thesynthesized recitation is generative such that a first synthesizedrecitation is different from a second synthesized recitation, each ofthe first synthesized recitation and the second synthesized recitationbased on the sequence of text.
 9. A method, comprising: determining,from a plurality of audio samples including human speech, alignmentsbetween text of the plurality of audio samples, a duration of theplurality of audio samples, and at least one of a pitch of the audiosamples or an energy of the audio samples; generating an alignmentdistribution based, at least in part, on the alignments; determining asoft alignment between a first text sequence from the text andmel-frames of the alignment distribution, the soft alignment normalizingprobability distributions for the alignments across the duration of theplurality of audio samples; and determining a hard alignment between thefirst text sequence and the mel-frames of the alignment distribution,the hard alignment concentrating the probability distributions from thesoft alignment to a symbol for the alignments across the duration of theplurality of audio samples; determining one or more vectors, based onthe hard alignment, corresponding to one or more speakercharacteristics; receiving a second text sequence; and generating, basedat least in part on the second text sequence and the one or morevectors, a synthetic audio clip corresponding to the second textsequence.
 10. The method of claim 9, wherein the alignment is based, atleast in part, on an alignment matrix with a beta-binomial distribution.11. The method of claim 9, wherein an encoder and a decoder forgenerating the synthetic audio clip operate in parallel.
 12. The methodof claim 9, further comprising: generating a second synthetic audio clipfrom the second text sequence, the second synthetic audio clip beingdifferent from the first synthetic audio clip.
 13. The method of claim9, further comprising: generating at least one of a phonemedistribution, a pitch distribution, or an energy distribution; andsampling from at least one of the phoneme distribution, the pitchdistribution, or the energy distribution.
 14. A processor, comprising:one or more processing units to: receive an audio clip of human speechrepresented as a mel-spectrogram; determine an alignment matrix for theaudio clip normalized to a probability distribution; apply, to thealignment matrix, a prior distribution to exclude pairs of phonemes andmel-frames in the audio clip from the alignment matrix; determine, fromthe alignment matrix, an alignment for a first text sequence within theaudio clip and a plurality of mel-frames representing a duration of theaudio clip; determine one or more distributions for a pitch and anenergy associated with the audio clip; receive a second text sequence;and generate a second audio clip of the second text sequence based, atleast in part, on the alignment and the one or more distributions. 15.The processor of claim 14, wherein the one or more processing units arefurther to implement an encoder and decoder for generating the secondaudio clip, wherein the encoder and decoder operate in parallel.
 16. Theprocessor of claim 14, wherein the one or more processing units arefurther to: fit, onto the alignment matrix, a beta-binomialdistribution.
 17. The processor of claim 16, wherein the beta-binomialdistribution excludes pairs of phonemes and mel-frames outside of aspecified range.
 18. The processor of claim 14, wherein the one or moreprocessing units are further to generate a third audio clip from thesecond text sequence, the second audio clip being different from thethird audio clip.